The present invention relates to a system and method for verifying and editing optically imaged responses received from a response provider.
The scoring of test question responses that cannot be scored by machine is typically carried out manually or by presentation on a computer monitor. Manual scoring involves a human manually scoring a physical test question response sheet. Scoring by presentation to a human of the test question responses using a computer involves scoring an electronic representation of test question response or responses presented to a scorer via a computer monitor or other machine that can be programmed to manipulate symbols. It is the latter scoring procedure to which the present invention relates.
In order to present a test question response to a scorer viewing a computer monitor, several preparation steps typically occur to enable the scorer to receive the test question response, view it, score it and record the score with the necessary precision, speed and accuracy required in the test scoring industry. With test processing that analyzes optically imaged test question responses, scanners, facsimile machines and other optical imaging devices known to those skilled in the art are used to create an electronic image of the test question response that was provided by a test taker. The electronic images may be broken down into smaller images generally referred to as image clips. The electronic images and image clips may be stored in a computer or other storage media known to those skilled in the art. The electronic images and image clips are typically extracted to data using well known and commercially available optical character recognition software, image character recognition software and other similar computer programs. The data can then be utilized in a number of ways to aid the test scoring process.
While character recognition software programs have improved greatly, a need still exists to verify the results of the image conversion. Verification is especially important in the test scoring industry where test takers"" scores may depend on the accuracy of the data extracted from the electronic images of the test question responses. Most character recognition software programs provide an automatic verification process performed electronically; however, even those results often are subject to a relatively high rate of error. In many situations, human editors use computer monitors at remotely-located geographic locations to verify the accuracy of the data extracted from the electronic images. A workflow system may be utilized to present to the editors data or images, or both, from which they are to verify accuracy.
Several problems using human editors to verify converted images have been identified. Editors often lack particular skills desired to analyze all types of data that may have been extracted from images. For example, if the test taker used Chinese characters to indicate his/her test question responses, editors not fluent in Chinese would find it difficult to analyze the information presented to them. Similarly, editors not comfortable with the numerous and complex symbols used on math tests may find it difficult to analyze the math test question responses presented to them.
The slow speed at which editors typically have analyzed the data extracted from images has also been a problem. Editors were randomly presented with data and/or images in need of analysis. For instance, an editor might be first presented with the image or the image and data of the test taker""s first name. Rather than presenting a last name or middle initial to the editor next, the editor might be presented with the image or the image and data of a math test question response. The random nature of the materials presented to editors slowed their process in verifying the data extracted.
The present distributed test processing workflow system facilitates accurate, consistent and high quality image capture, conversion to data and verification of test question responses provided by a test taker.
The present verification editing method divides test question response images into smaller images or xe2x80x9cimage clipsxe2x80x9d for conversion to data using well known image recognition software. In the method, it is determined whether the particular data obtained from the image clips requires verification. If verification is required, information related to each image clip is processed against particular appropriate skills of human editors in order to select human editors having particular appropriate skills. Selected image clips are then recompiled into coherent image portions of the test question response and selectively presented, with or without corresponding data, to the human editor for analysis. Finally, the decisions of the human editor are electronically recorded.